<<<<<<< HEAD

仅使用前视投影得到的效果更好？可能是网络的规模不够大

尝试仅使用前视投影+sn

删除前视投影仅保留pc分支：

loss非常不稳定生成图像效果差

将pc分支的浅层特征经过down和res分支

另外将pc特征加入到discriminator中或者从discriminator中取消投影输入

SN: spectral norm

在discriminator中删除conditional输入之后， feature matching loss和vgg loss的初始值变得非常高，这可能是因为没有conditional输入之后，discriminator在开始接受到的输入就完全是混乱的初始生成图像没有与真实图像的feature比较类似的条件输入；但是loss下降的趋势还比较稳定(即没有造成训练过程的不稳定)，后期也下降到和有conditional输入的discriminator一样的情况了

最终的测试结果没有加入conditional输入好

目前来看Generator输入的上采样图像的作用比较大

静止场景：

0016 0017 0019（步行街场景行人多光照复杂）

尝试noVGG 效果很差

尝试pixel shuffle 解决棋盘格效应

object sensitive loss

尝试对gamma进行随机增强，生成的图像亮度更高，在50轮的时候视觉效果更好一些，但是没有最开始的点云分支好

尝试加入pointnet2的head，收敛速度变慢，训练集有一些图像效果还可以，对训练集拟合效果不错

激光雷达+图像去阴影

和相机重建出来的对比

将测试集换为tracking与object的差集效果不错

尝试在resblock前加入pc特征效果比resblock前加入的细节和边缘要好

对点云的intensity进行归一化

在Discriminator中取消SN后面的BN之后 Gan的Feature matching loss的量级下降了很大从[7,12]下降到了[1, 2]，这可能是因为SN实现了Lipschitz连续条件但是只使用SN时拟合效果非常差

无法复现之前pix2pix的效果了目前的改动：Gamma：无效 flip：无效 batchsize 有一些效果，现在感觉可能是数据集的问题

尝试直接用前视投影的点云提特征？

加入膨胀卷积之后效果很好

远处的弱化从loss上考虑

LiCAM segmentation 实验

Accumulating evaluation results... │tmpfs 38G 3.8G 34G 10% /run

DONE (t=0.06s). │/dev/sda2 439G 393G 24G 95% /

Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.590 │tmpfs 189G 736M 188G 1% /dev/shm

Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.882 │tmpfs 5.0M 0 5.0M 0% /run/lock

Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.646 │tmpfs 189G 0 189G 0% /sys/fs/cgroup

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.402 │/dev/sda1 511M 3.7M 508M 1% /boot/efi

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.689 │tmpfs 38G 28K 38G 1% /run/user/108

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.866 │tmpfs 38G 22G 16G 58% /run/user/1000

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.180 │tmpfs 38G 0 38G 0% /run/user/1001

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.613 │(base) ➜ sTrain-sTest_LiCAM_cas101_MS sudo kill 3771

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.625 │[sudo] password for bdbc201:

Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.453 │(base) ➜ sTrain-sTest_LiCAM_cas101_MS

Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.715 │(base) ➜ sTrain-sTest_LiCAM_cas101_MS du -sh

Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.907

LiCAM 测真实图像

DONE (t=0.20s). │/dev/sda2 439G 393G 24G 95% /

Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.613 │tmpfs 189G 736M 188G 1% /dev/shm

Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.885 │tmpfs 5.0M 0 5.0M 0% /run/lock

Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.689 │tmpfs 189G 0 189G 0% /sys/fs/cgroup

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.397 │/dev/sda1 511M 3.7M 508M 1% /boot/efi

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.737 │tmpfs 38G 28K 38G 1% /run/user/108

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.886 │tmpfs 38G 22G 16G 58% /run/user/1000

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.195 │tmpfs 38G 0 38G 0% /run/user/1001

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.646 │(base) ➜ sTrain-sTest_LiCAM_cas101_MS sudo kill 3771

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.662 │[sudo] password for bdbc201:

Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.458 │(base) ➜ sTrain-sTest_LiCAM_cas101_MS

Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.778 │(base) ➜ sTrain-sTest_LiCAM_cas101_MS du -sh

Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.932

真实图像测SPADE

Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.449

Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.678

Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.485

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.224

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.582

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.783

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.161

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.481

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.481

Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.254

Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.610

Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.816

真实图像测皮鞋pix2pixHD

Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.146

Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.301

Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.130

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.038

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.179

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.407

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.092

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.168

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.168

Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.043

Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.209

Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.442

真实图像测LiCAM

Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.388

Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.636

Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.407

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.151

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.511

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.809

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.152

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.429

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.431

Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.186

Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.553

Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.847

真实图像测SPADE-LICAM-Intensity

直接High res

Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.371

Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.586

Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.410

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.111

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.502

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.813

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.141

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.402

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.402

Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.137

Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.532

Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.858

Low res 2 High res

Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.365

Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.583

Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.399

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.112

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.496

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.791

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.142

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.398

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.398

Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.138

Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.530

Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.829

目前看应该是需要平滑的conditional input（这个地方可以abalation study）

真实图像测SAPDE-LICAM

Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.337

Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.550

Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.362

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.102

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.453

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.767

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.135

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.376

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.376

Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.128

Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.494

Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.814

真实图像测pix2pix

Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.007

Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.010

Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.010

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.004

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.015

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.003

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.003

Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.003

Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000

Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.003

Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.014

真实图像测cycle

None 没有结果

改用OMP多核实现之后生成1M个点可以从90s提高到10s

使用光栅化过程的zbuffer解决了深度遮挡的问题可以考虑不用光线追踪但是仍然是串行实现的

ray tracng 之后的结果会在属于同一个类别的大片联通区域产生孔洞，孔洞中的数值是与该区域的类别非常相似的数值。这在图像中无所谓，但是在semantic label里，相近id代表完全不同的类别，因此使用闭运算，先膨胀再腐蚀，可以很好的消除孔洞

闭运算后

闭运算前

**** SPADE 没有crop图像的上面120个pixel

U-Net或者hourglass 应用在point network中

点云segmentation 之后用结果对应图像的纹理做tracking

标定想通过射线投影模型，把方差较大的点投影会拟合的平面上但是由于x方向方差过大拟合的标定板投影实际上出现了仿射变换无法构成矩形棋盘

在分割棋盘格时，最开始是找Z轴方向密度变化最大的一个点作为分割点，但是由于horiozon扫出来的点扫镖器边缘密度不均匀，很容易在棋盘中间造成分割点。为了解决这个问题，每次找到Topk个可能的分割点，计算每个点与棋盘格长度的差，找到差距最小的那个点作为真正的分割点，这样可以保证这个点是可以覆盖整个棋盘的

尝试了用求得的外参反向计算内参，但是发现求得的结果还是和内参输入是一样的，这可能是因为固定了错误的内参，反向计算的内参都是在过拟合这个外参

对比手工标注和棋盘格标到相同的RMSE所需要的标定板摆放次数

** feature map在空间上进行泊松融合

轨迹叠加追踪

* 寻找最优的摆放位置，可以先尝试暴力搜索，在给定的一组lidar-camera pair中找到一组子集，使得准确率最高

扰动LiDAR点云的坐标系，这样就构成了不同位置的点云-图像pair

legoloam 在狭窄走廊+行人遮挡会出现degeneration; 这个degeneration还和建图开始的位置有关, 因为不同的建图开始位置会导致不同的局部地图, 从而对mapoptimization的局部地图匹配造成影响;这个问题在lio-sam中更严重

BEV

BBA 在KITTI鸟瞰图上直接检测效果特别差

AP@0.3只有0.14512388417054484

AP@0.5只有0.02

MotionNet

nuscene part1

train

1400短序列

val 170

Mean pixel classification accuracy: 0.9270618313934371

mean cat accuracy of Bg: 0.9455821795090201

mean cat accuracy of Vehicle: 0.5206497586391847

mean cat accuracy of Ped: 0.44901742677048573

mean cat accuracy of Bike: 0.0

mean cat accuracy of Others: 0.588819095477387

mean instance acc: 0.5008136920792154

test 500

Mean pixel classification accuracy: 0.9226152206242199

mean cat accuracy of Bg: 0.9450818408534926

mean cat accuracy of Vehicle: 0.574332482538324

mean cat accuracy of Ped: 0.5306154084638041

mean cat accuracy of Bike: 0.0

mean cat accuracy of Others: 0.5026899037205982

mean instance acc: 0.5105439271152438

livox horion的点云时间分布规律

每一帧的相对时间分布是有相位差的, 比如前25%收到的点, 并不在帧数据的前25%位置, 而是在整个点云中滑动. 这个滑动可能通过运动模型计算出来.

=======